---
id: testing-kafka-go-api-with-opentelemetry-tracetest
title: Testing Kafka in a Go API with OpenTelemetry and Tracetest
description: Quick start how to configure two Go APIs to communicate via Kafka. They use OpenTelemetry instrumentation with traces, Jaeger as a trace data store, and Tracetest for enhancing your E2E and integration tests with trace-based testing.
hide_table_of_contents: false
keywords:
  - tracetest
  - trace-based testing
  - observability
  - distributed tracing
  - testing
  - testing event driven systems
  - testing event driven apps
  - event driven testing
  - event driven observability
  - testing kafka
  - testing kafka queues
  - testing kafka messages
  - opentelemetry
image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg
---

:::note
[Check out the source code on GitHub here.](https://github.com/kubeshop/tracetest/tree/main/examples/quick-start-go-and-kafka)
:::

[Tracetest](https://tracetest.io/) is a testing tool based on [OpenTelemetry](https://opentelemetry.io/) that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

## Sample Go APIs with OpenTelemetry Collector, Jaeger and Tracetest

This is a simple quick start on how to configure two Go APIs to communicate via Kafka. They use OpenTelemetry instrumentation with traces, Jaeger as a trace data store, and Tracetest for enhancing your E2E and integration tests with trace-based testing.

## Prerequisites

**Tracetest Account**:

- Sign up to [`app.tracetest.io`](https://app.tracetest.io) or follow the [get started](/getting-started/overview) docs.
- Have access to the environment's [agent API key](https://app.tracetest.io/retrieve-token).

**Docker**: Have [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed on your machine.

## Run This Quckstart Example

The example below is provided as part of the Tracetest project. You can download and run the example by following these steps:

Clone the Tracetest project and go to the Node.js Quickstart with Manual Instrumentation:

```bash
git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/quick-start-go-and-kafka
```

Follow these instructions to run the quick start:

1. Copy the `.env.template` file to `.env`.
2. Fill out the [TRACETEST_TOKEN and ENVIRONMENT_ID](https://app.tracetest.io/retrieve-token) details by editing your `.env` file.
3. Run `docker compose run tracetest-run`.
4. Follow the links in the output to view the test results.

Follow along with the sections below for an in detail breakdown of what the example you just ran did and how it works.

## Project Structure

The quick start Kafka + Go project is built with Docker Compose and contains the Tracetest Agent, Jaeger, OpenTelemetry Collector and a both Go apps.

The `docker-compose.yaml` file in the root directory of the quick start runs the Go apps, Jaeger, OpenTelemetry Collector, and the [Tracetest Agent](/concepts/agent) setup.

## Go Producer API

The Go API is a simple HTTP server, [contained in the `main.go` file](https://github.com/kubeshop/tracetest/blob/main/examples/quick-start-go-and-kafka/producer-api/main.go).

The OpenTelemetry tracing is [contained in the `./telemetry/telemetry.go` file](https://github.com/kubeshop/tracetest/blob/main/examples/quick-start-go-and-kafka/producer-api/telemetry/telemetry.go).
Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.

Traces will be sent to either the `grpc` endpoint. The hostname and port as seen in the env section of the `producer-api` in the `docker-compose.yaml` is:

- `OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317`

The server starts by running the `main.go` file as you can see [in the `Dockerfile`](https://github.com/kubeshop/tracetest/blob/main/examples/quick-start-go-and-kafka/producer-api/Dockerfile).

[The `docker-compose.yaml` contains one service](https://github.com/kubeshop/tracetest/blob/3caff14247d745035f7cbbb68a7208f6da4e51fd/examples/quick-start-go-and-kafka/docker-compose.yaml#L3C3-L21C1) for the Go Producer API.

```yaml
producer-api:
  image: quick-start-producer-api
  platform: linux/amd64
  build: ./producer-api
  extra_hosts:
    - "host.docker.internal:host-gateway"
  ports:
    - 8080:8080
  environment:
    - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
    - OTEL_SERVICE_NAME=producer-api
    - KAFKA_BROKER_URL=kafka:9092
    - KAFKA_TOPIC=messaging
  depends_on:
    otel-collector:
      condition: service_started
    kafka:
      condition: service_healthy
```

## Go Consumer Worker

The Go API is a simple HTTP server, [contained in the `main.go` file](https://github.com/kubeshop/tracetest/blob/main/examples/quick-start-go-and-kafka/consumer-worker/main.go).

The OpenTelemetry tracing is [contained in the `./telemetry/telemetry.go` file](https://github.com/kubeshop/tracetest/blob/main/examples/quick-start-go-and-kafka/consumer-worker/telemetry/telemetry.go).
Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.

Traces will be sent to either the `grpc` endpoint. The hostname and port as seen in the env section of the `consumer-worker` in the `docker-compose.yaml` is:

- `OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317`

The server starts by running the `main.go` file. As you can see [in the `Dockerfile`](https://github.com/kubeshop/tracetest/blob/main/examples/quick-start-go-and-kafka/consumer-worker/Dockerfile).

[The `docker-compose.yaml` contains one service](https://github.com/kubeshop/tracetest/blob/f5d7cb6747813838623d1a61424788e12f21cece/examples/quick-start-go-and-kafka/docker-compose.yaml#L22-L39) for the Go Producer API.

```yaml
consumer-worker:
  image: quick-start-consumer-worker
  platform: linux/amd64
  build: ./consumer-worker
  extra_hosts:
    - "host.docker.internal:host-gateway"
  environment:
    - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
    - OTEL_SERVICE_NAME=consumer-worker
    - KAFKA_BROKER_URL=kafka:9092
    - KAFKA_TOPIC=messaging
  depends_on:
    otel-collector:
      condition: service_started
    kafka:
      condition: service_healthy
    producer-api:
      condition: service_started
```

## Running the Go Apps, Kafka, and Tracetest

Configure the `.env` like shown below.

```bash
# Get the required information here: https://app.tracetest.io/retrieve-token

TRACETEST_TOKEN="<YOUR_TRACETEST_TOKEN>"
TRACETEST_ENVIRONMENT_ID="<YOUR_ENV_ID>"
```

To start the full setup, run the following command:

```bash
docker compose run tracetest-run
```

This will:
1. Start the Go apps, Kafka, the OpenTelemetry Collector, and send the traces to Jaeger.
2. Start the Tracetest Agent.
3. Configure the tracing backend and create tests in your environment.
4. Run the tests.

Follow the links in the output to view the run group and the test result on Tracetest.

## Manually Running the Tracetest Tests

Open the URL and start creating tests! Make sure to use the `http://producer-api:8080/publish` URL in your test creation, because your Go app and Tracetest Agent are in the same network. And, use the [Kafka trigger](http://localhost:3000/web-ui/creating-tests-kafka) for the Consumer Worker.

Here are two sample tests you can get started with quickly.

👉 [Testing the Consumer](https://github.com/kubeshop/tracetest/blob/main/examples/quick-start-go-and-kafka/resources/test.yaml)

👉 [Testing the Producer](https://github.com/kubeshop/tracetest/blob/main/examples/quick-start-go-and-kafka/resources/test-producer.yaml)


## Learn More

Feel free to check out our [examples in GitHub](https://github.com/kubeshop/tracetest/tree/main/examples) and join our [Slack Community](https://dub.sh/tracetest-community) for more info!
